Methods and systems for complexity-based segmentation refinement

ABSTRACT

Embodiments of the present invention comprise methods and systems for image complexity estimation and selective complexity-based image processing.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.10/882,478, entitled “Methods and Systems for Complexity Estimation andComplexity-Based Selection,” invented by John Dolan, Toyohisa Matsuda,Ahmet Mufit Ferman & Richard Campbell, filed on Jun. 30, 2004.

BACKGROUND OF THE INVENTION

Digital images and documents may contain many elements or content typesincluding text, halftone, graphics, bitmap images, variations thereofand other elements. When rendered to a display or a printer, each ofthese elements may be processed in a different way to optimize thequality of the presented output. This differential processing requiresthat the image be segmented into elements or content types. This istypically performed by computing a so-called segmentation map from adigital image of a document page. Often this reduces to a problem ofpixel or region classification, since the set of element types orcontent types is known a priori. Given the segmentation map of an inputpage, each content type region can then be optimally processed accordingto the requirements of its corresponding elements or content type.

In some known methods, as shown in FIG. 1, object data for a renderingjob is received 10. This data 10 is typically in the form of printer joblanguage commands or graphics engine rendering commands such as HPGLcommands, PCL commands, GDI commands or others. These commands identifythe content type for the graphic elements they define and thisinformation can be easily extracted from the command data 10 to identify12 the content types in the document. Once the content types areidentified 12, the configuration of the objects can be analyzed 14 tohelp evaluate document complexity. A complexity factor is calculated 16from this data. While these techniques work well for document data thatis compartmentalized into command structures, such as object-basedcommands, it is of no use on bitmap data, raster data and other forms ofnon-object-based data. Additionally, the available methods have not beencombined with processing algorithm data to create an algorithm-relatedcomplexity factor.

BRIEF SUMMARY OF THE INVENTION

Embodiments of the present invention comprise systems and methods forestimating document image complexity and using the complexity estimationas a basis for image processing decisions such as selecting a process.

The objectives, features, and advantages of the invention will be morereadily understood upon consideration of the following detaileddescription of the invention taken in conjunction with the accompanyingdrawings.

BRIEF DESCRIPTION OF THE SEVERAL DRAWINGS

FIG. 1 is a chart showing prior art complexity calculation methods;

FIG. 2 is a graph showing the effect on image quality ofsegmentation-based enhancement as image complexity increases;

FIG. 3 is a chart showing a raster-based complexity determination methodof embodiments of the present invention;

FIG. 4 is a chart showing a raster-based complexity determination methodof embodiments of the present invention wherein algorithmcharacteristics are employed;

FIG. 5 is a chart showing a raster-based complexity determination methodof embodiments of the present invention wherein transform domain data isemployed;

FIG. 6 is a chart showing an object-based complexity determinationmethod of embodiments of the present invention wherein algorithmcharacteristics are employed;

FIG. 7 is a diagram showing embodiments of the present invention thatdetermine segmentation-based complexity measures and use these measuresto control image enhancement;

FIG. 8 is a diagram showing embodiments of the present invention thatdetermine segmentation-based complexity measures and use these measuresto control image enhancement and to control a combination of global andsegmentation-based enhancement;

FIG. 9A is a diagram showing embodiments of the present invention thatuse coarse segmentation to determine complexity measures and laterperform a more refined segmentation if needed;

FIG. 9B is a diagram showing embodiments of the present invention thatuse coarse segmentation to determine complexity measures and laterperform a more refined segmentation and a refined complexitydetermination if needed;

FIG. 10 is a diagram showing embodiments of the present invention thatdetermine segmentation-based complexity measures for successive imagesections and employ a cumulative complexity measure to control imageenhancement;

FIG. 11 is a diagram showing embodiments of the present invention thatdetermine complexity measures from page-related features without a needfor segmentation during complexity determination;

FIG. 12 is a diagram showing embodiments of the present invention thatdetermine complexity measures from page-related features without a needfor segmentation during complexity determination and which calculatecomplexity using successive image sections;

FIG. 13 is a diagram showing embodiments of the present invention thatdetermine complexity measures and use these measures as a gain controlon enhancement processes;

FIG. 14 is a table showing a multiple filter configuration method;

FIG. 15 is a diagram showing embodiments of the present invention thatdetermine region-specific complexity measures;

FIG. 16 is a diagram showing embodiments of the present invention thatdetermine multiple complexity measures using multiple segmentationmethods;

FIG. 17 is a diagram showing embodiments of the present invention thatdetermine multiple complexity measures using multiple segmentationmethods and perform multiple tuned enhancements, which may be combinedinto a final enhanced image;

FIG. 18 is a diagram showing embodiments of the present invention thatdetermine multiple complexity measures using multiple, initial, coarsesegmentation methods; and

FIG. 19 is a chart showing calculated complexity measures plotted with asubjective complexity determination by human observers.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The described embodiments of the present invention will be bestunderstood by reference to the drawings, wherein like parts aredesignated by like numerals throughout. The figures listed above areexpressly incorporated as part of this detailed description.

It will be readily understood that the components of the presentinvention, as generally described and illustrated in the figures herein,could be arranged and designed in a wide variety of differentconfigurations. Thus, the following more detailed description of theembodiments of the methods and systems of the present invention is notintended to limit the scope of the invention but it is merelyrepresentative of the exemplary embodiments of the invention.

Many of the embodiments described in this description are examples fromthe field of copying and reprographics, where segmentation is used toimprove image quality through optimized, selective enhancement of pageelements. However, these are only exemplary embodiments that should inno way limit the scope of the present invention from extending to otherdomains that exploit segmentation-guided processing, such ascompression, archiving, retrieval and others.

Elements of embodiments of the present invention may be embodied inhardware, firmware and/or software. While exemplary embodiments revealedherein may only describe one of these forms, it is to be understood thatone skilled in the art would be able to effectuate these elements in anyof these forms while resting within the scope of the present invention.

Little, if any, attention has been given to the problem of determiningwhen a segmentation procedure is appropriate or not—in other words,determining the reliability and/or the cost-effectiveness of thesegmentation.

Embodiments of the present invention address the problem of computingthe reliability and/or benefit of segmentation algorithms and theproblem of adjusting the ensuing processing accordingly. Some of theseembodiments may explicitly identify cases in which the segmentationprocedure is either highly error prone or too costly to complete.

In the field of printing and reprographics and many other applications,the image quality benefit of segmentation attains a maximum formoderately complex documents. As illustrated in FIG. 2, the benefitincreases as page layout complexity increases only up to some point 2.It thereafter decreases due to an increased probability of segmentationerrors and the increased difficulty of the segmentation task itself.More significantly such errors often lead to noticeable artifacts in theoutput page that can negate any benefit derived from the segmentation.

In FIG. 2, the approximate point at which the benefit of segmentedenhancement dips below that of “unsegmented” or global enhancement isindicated by the intersection of the image quality curve and a verticalline 4. This point depends both on the complexity of the page layout andthe exact nature of the segmentation algorithms. Typically, this pointis determined experimentally.

In order to limit segmentation to only beneficial cases or to adjustsegmentation for more optimal processing, embodiments of the presentinvention may compute a measure of page layout complexity. Such ameasure may determine the overall complexity of a document page, whichmay include page layout.

The terms “page complexity,” “page layout complexity” and “imagecomplexity” are used synonymously throughout the specification andclaims of this application. All these terms refer to the quantity, size,geometric relationship and other attributes of document elements orcontent types such as areas of text, graphics, halftone elements, bitmapelements and other elements.

A complexity measure may also take into account the error likelihoods ofalternative processing techniques, such as segmentation algorithms orsets of possible algorithms. Once determined, this measure may be usedto determine whether or not a processing step, such as segmentationshould be applied in subsequent enhancement processing, thereby avoidingenhancement artifacts. Complexity measures may also be used to determinethe type or extent of processing, including but not limited tosegmentation, used for a particular application.

The term “content region” or “region” may refer to an area or group ofareas that contain a single content type. For example, a halftone regionmay contain all of the halftone elements of an image, which may bescattered across several contiguous or non-contiguous areas.

The terms “raster data,” “bitmap data,” and “pixel data,” as used in thespecification and claims of this document, refer to document or imagedata that define a document or image relative to basic picture elementsor pixels. These terms and the more generic term, “non-object-based” maybe used to refer to image or document data that relates directly tobasic image pixels rather than objects that contain a plurality ofpixels in a pre-defined configuration. Non-object-based data comprisestreaming data and transform domain data, such as transformcoefficients, that are related to pixel-based images or documents.Printer job language commands and graphics engine rendering languagecommands are typically object-based data.

The effectiveness of image processing techniques, such as pagesegmentation, depends both on the complexity of the page and its layoutand the exact nature of the processing (i.e., segmentation) algorithm.Factors may include resource costs in terms of time and memory, and thesuccess/failure rate of the classification task itself. However, in thecase of segmentation, the benefit of segmentation tends to decreaseafter a certain level of layout complexity is reached, regardless of theparticular algorithm. This may occur simply because the classificationtask itself becomes increasingly difficult. This may also occur whensegmentation area boundaries become so numerous that boundary artifactssignificantly degrade image quality. Regardless of the reasons for thediminishing returns, the benefits of segmentation begin to decrease aspage or page layout complexity increases beyond a certain level.

Thus for a given process, such as a segmentation algorithm or group ofalgorithms, it is possible to determine, in terms of page layoutcomplexity, when application of the particular algorithm or group willbe beneficial and when it will not improve image quality.

Some embodiments of the present invention may be explained withreference to FIG. 3. In these embodiments, non-object-based (NOB) rasterdata for a page or section is received and read 20. This raster NOB datais then analyzed to identify content types 22 or a likelihood thatcontent types exist in the page or section. Various content types mayaffect the final complexity factor in different ways. In this step, eachtype or a likelihood of its existence is identified 22. As content typesare identified, elements of each type may be associated with areas ofthe image and the accumulated areas for a specific content type may beorganized into a content type region 24. Characteristics of each contenttype region may be determined by analysis of these regions. A complexityfactor may then be calculated 26 based on region characteristics as wellas other data.

A complexity factor will typically relate to content type regioncharacteristics, such as region size, shape, quantity, density, regionalgeometry, region contiguity and other region characteristics. Acomplexity factor may also relate to other page, section or imagecharacteristics as well as characteristics of a process to be selectedor controlled with reference to the complexity factor. Complexityfactors that are calculated from raster data may be referred to asraster-related complexity factors.

In some embodiments, a complexity factor may relate to a process oralgorithm on which the complexity factor will have an effect. Some ofthese embodiments are illustrated in FIG. 4. In these embodiments,raster page or section data is read 30 and content types are identified32. Content regions are identified and region characteristics aredetermined 34. Characteristics of an algorithm or process that will beaffected by the complexity factor may also be determined 36 and used inthe complexity factor calculus. These algorithm characteristics maycomprise the reliability of a specific algorithm in relation to acontent type or types, or other data. Once algorithm and regioncharacteristics are determined, an algorithm-related complexity factorcan be calculated 38.

In some embodiments of the present invention, as illustrated in FIG. 5,a complexity factor may be calculated in relation to transform domaindata. Transform domain data may comprise transform coefficients orrelated data. A transform commonly used in image processing is theDiscrete Cosine Transform (DCT), however many other transforms may beused in embodiments of the present invention.

In these embodiments the transform domain data is read 40 and used toidentify content types 42. Content type regions are then constructed andregion characteristics are determined 44. In some embodiments, anoptional step of determining algorithm or process characteristics may beperformed 46. Once their characteristics are determined, they may beused to calculate a complexity factor 48. This complexity factor may bean algorithm-related complexity factor. Complexity factors that arecalculated with transform domain data may be referred to astransform-data-related complexity factors.

In some embodiments of the present invention, as illustrated in FIG. 6,rendering stream object data may be read 50 and used to identify contenttypes 52 as is done in some known methods. This object data may be usedto determine content region characteristics 54. Characteristics of analgorithm or process to be affected by the complexity factor may also bedetermined 56. A complexity factor that is dependent on both the imagecomplexity and the algorithm or process may then be calculated 58. Acomplexity factor that is related to both the image complexity and theeffect of image complexity on the result of an algorithm or process maybe referred to as process-effect complexity factor.

Some embodiments of the present invention may be described in relationto FIG. 7. The input document image 120 is segmented 110 into itsconstituent regions, and, in some embodiments, a segmentation map thatidentifies the content type or types for each pixel is generated. Thismap is subsequently used to measure the complexity 112 of the document,taking into account the layout projected by the segmentation map. Insome embodiments, the properties of the particular segmentation methodthat may be employed can have an effect on the complexity estimation.The resulting complexity value Cx 114 provides an estimate of howreliable the segmentation map is, and how appropriate asegmentation-based, region-specific enhancement approach would be forthe input document. In some embodiments, if Cx 114 exceeds apredetermined threshold, the segmentation map is discarded, and astandard global enhancement 116 is applied to the document to avoidpotential artifacts in the output image. Otherwise, the document isenhanced 118 based on the segmentation map, enabling an optimizedprocess to be performed on each content type.

Some embodiments of the present invention may be described in relationto FIG. 8, which illustrates exemplary embodiments comprising complexitymeasures that can be utilized for segmentation-based documentenhancement. In these embodiments, segmentation-based enhancement 126and global enhancement 124 can be performed independently. The resultsmay then be combined 28 based on the computed complexity 114 of thedocument. Typically, an input image 120 is processed using asegmentation algorithm 122 thereby creating a segmentation map 125. Thismap 125, may be used to perform segmentation-based enhancement. Inputimage 120 may also be processed directly by performing globalenhancement 124. These enhancement processes may take place in parallel,series or otherwise.

After map 125 is created, the complexity of the image 120 is measured130 and a complexity value 114 is calculated. When document complexityis low, segmentation-based enhancement 126 may be weighed more heavily,whereas global enhancement 124 may be given more emphasis for a complexdocument. The two enhancements may be combined 128 in various ways. Forexample, a simple linear combination method, such as(1−C_(x))·SegEnh+C_(x)·GlobalEnhcan be employed when the complexity measure is a scalar. Here SegEnh andGlobalEnh denote segmentation-based enhancement 126 and globalenhancement 124, respectively. Alternatively, the combination rules maybe region-driven; i.e. they may be based on the performance of thesegmentation and enhancement algorithms for detection and improvement ofvarious types of content. If the complexity of the document is definedon a region-by-region basis rather than the entire page, regional,global or default enhancement methods may be preferred in areas wheresegmentation confidence is low.

In further embodiments, illustrated in FIG. 9A, an additionalsegmentation modification may be used. In these embodiments anadditional segmentation refinement step is used when the complexity ofthe document is found to be below a predetermined threshold, σ or foundto meet some other criterion. This approach allows a coarse initialsegmentation map to be generated first and used for complexity analysis,thereby reducing the computational requirements of the implementation.In these embodiments an input image 120 is processed. Initialsegmentation is performed 132 and an initial segmentation map 134 iscreated. As in other embodiments, the complexity of the image ismeasured 136 and a complexity value 138 is calculated. When thecomplexity value 138 is greater than a threshold value 140, or thecomplexity value meets some other criterion, global enhancement 142 maybe performed yielding a globally enhanced image 150. When the complexityvalue 138 is less than the threshold value 140, or meets some othercriterion, a finer segmentation process may be performed 144 and arefined segmentation map 146 may be created.

The initial segmentation map 135 may be passed to the refinedsegmentation processor for use in the refined process. In alternativeembodiments, the initial map may not be used and an independent refinedsegmentation process may be performed.

The image 120 may then be enhanced using a segmentation-basedenhancement 148 based on the refined map 146. This part of the processwill yield an image 152 enhanced by a refined segmentation-basedenhancement.

In some embodiments of the present invention, as illustrated in FIG. 9B,an alternative, refined segmentation approach may be used. In theseembodiments, an input image 120 is processed and initial segmentation isperformed 242 yielding an initial segmentation map 244. This initial map244 is used to measure 246 the complexity of the image 120. A complexityvalue 250 is determined by this process. If the complexity value meets agiven criterion, global enhancement 252 is performed and a globallyenhanced image 251 results. If that criterion is not met or analternative criterion is met, a finer segmentation process 254 may beperformed. This process may use the initial segmentation map 245 as abasis for its refined process or may perform a finer segmentationprocess 254 independent of the initial segmentation process 242. Thisfiner segmentation process 254 may result in a finer segmentation map256.

Once the refined map 256 has been generated, the image complexity mayagain be measured 258 based on the refined map 256 and a refinedcomplexity value 260 may be determined. If this refined complexity valuemeets a criterion 262, such as exceeding a threshold value, a globalenhancement 252 may be performed. If that criterion is not met or analternative criterion is met, an alternative enhancement process 264 maybe performed thereby yielding an enhanced image 266.

In the previously-described embodiments of the present invention, thecomplexity measure may be computed using the segmentation map of theentire document image. In alternative embodiments, described withreference to FIG. 10, the complexity measure may be calculated andupdated or accumulated progressively using image sections or strips. Forthe purposes of this specification and claims, the term section is usedto refer to any sub-division or portion of a document, document image orimage. A section may be formed by dividing the image geometrically, bycolor characteristics, by transform values or by some other method. Theterm “strip” refers to an elongate geometric portion of an image,typically extending from one end or side of a page to an opposite end orside. The term “strip” is comprised within the meaning of the broaderterm “section.” In these embodiments, an input image 120 is processedincrementally by section or strip. This can be an iterative process inwhich a first section is analyzed 60 and segmentation is performed 162thereon. Once segmentation has been performed on one section, acumulative section segmentation map 182 is created. The complexity ofthe image section is also calculated 64 and a complexity value ormeasure 168 is established 166. If the complexity value 168 exceeds 170a threshold value, the segmentation process is terminated and globalenhancement is performed 172.

If the complexity value 168 is less than 170 a threshold value, anotherimage section is processed 160. When subsequent sections are analyzed,the segmentation map is updated 180 and a complexity measure iscalculated 64 for that section. An accumulated complexity measure 168 isalso updated 166 to reflect information gathered in the new section. Ifthe accumulated complexity value 168 exceeds 170 the threshold value,global enhancement 172 is performed. If the accumulated complexity value168 remains below the threshold value, another section is processed 160.This process may continue until the complexity measure 168 exceeds thethreshold value or until the entire page is processed 176. If the entirepage is processed 176 without exceeding the threshold value, the entireimage is processed using a tuned, segmentation-based enhancement 1714.In some embodiments, non-overlapping image strips which comprise n rows,with n≧1 may be used.

In further embodiments of the present invention, shown in FIG. 11, pagecomplexity may be determined without an explicit segmentation map. Inthese embodiments, other image or page features directly extracted fromthe input document, such as, but not limited to, histogram analysis andtransform domain processing may be utilized to estimate the complexityof the document. This initial analysis and complexity estimate issubsequently employed to determine whether it is worthwhile to generatea segmentation map and perform tuned enhancement on the document. Inthese embodiments an input image 120 is processed and analyzed tocompute image or page features 184. Page complexity is estimated 186based on these features and a complexity value or measure 188 iscalculated. When the complexity measure 188 exceeds a threshold value190, global enhancement 192 is performed. When the complexity measure isless than the threshold value, segmentation is performed 194 and asegmentation map is created 196. The image is then enhanced 198 based onthe segmentation map.

Further embodiments of the present invention may be explained withreference to FIG. 12. In these embodiments, an input image 120 isprocessed in strips or sections. A first strip or section is analyzed190 and the complexity of that strip or section is determined 192 usingpage-related features extracted from the input image 120. In theseembodiments, page-related features include, but are not limited to,histogram analysis and transform domain processing. A cumulative pagecomplexity measure 196 is stored 194. If the cumulative page complexitymeasure 196 exceeds a threshold value 198, global image enhancement 200may be performed. If the cumulative page complexity measure 196 is lessthan the threshold value 198 and the entire document has not yet beenprocessed 202, the next image strip or section is analyzed 190. Thecomplexity of the next image strip or section is used to update 194 thecumulative page complexity measure 196 and this measure 196 is againcompared to a threshold value 198. If the measure 196 exceeds thethreshold value 198, global processing may be immediately performed forthe entire image 200. If the threshold value is not exceeded 198, andthe entire document has not yet been processed 202, another strip orsection is processed as explained above.

This iterative process is continued until the threshold is exceeded 198or the entire image is processed 202. If the entire image is processedwithout exceeding 198 the threshold value, the segmentation is performedon the entire image 204 and the image is enhanced using asegmentation-based enhancement 208. In some embodiments, a segmentationmap 206 is used in this final segmentation-based process, however, thesegmentation map does not need to be created unless the complexitymeasure 196 remains below the threshold value for the entire document.

In some embodiments of the present invention, an estimated complexityvalue may be used to adjust the level of enhancement that is applied toan input image. In these embodiments, the complexity value serves as atype of gain control and may determine how aggressivelysegmentation-based enhancement will be exercised. Rather than forcing abinary decision between two distinct types of enhancement (i.e., globalvs. segmentation-based), the strength of the enhancement may beregulated. This may be performed on a continuous scale, on a step-wisebasis or by some other adjustment scheme.

Typical embodiments may be described with reference to FIG. 13. In theseembodiments, an input image 20 is analyzed and segmentation 220 isperformed thereon whereby a segmentation map 222 is created. Thecomplexity of the image is then measured 224 and a complexity value isdetermined 226. Based on this complexity value 226, a tuned enhancementmay be performed 228 wherein the level of enhancement is variedaccording to the complexity value 226.

Adjustment of segmentation-based enhancement can be performed in variousways. FIG. 14 shows an exemplary embodiment comprising multiple filtertypes with different coefficients. These filters and coefficients may beutilized based on the value of the complexity estimate as shown.

Alternatively, the same set of enhancement filter coefficients can bemodified based on the value of the complexity estimate C_(x); forexample, filter responses may be adjusted according to C_(x), so that asdocument complexity increases, the filters tend to more conservativeenhancement. In an alternative implementation, the complexity estimatecan be used to combine the enhancement filter coefficients orenhancement results for various region types (e.g., text, halftone,etc.), to ensure that overaggressive processing is not applied to theinput image.

In some previous embodiments of the present invention the complexity ofa document has been defined for an entire image. In alternativeembodiments, the complexity measure may be multi-valued. In someembodiments, the complexity measure may be represented by a vector whosecomponents reflect the complexity of specific content regions in thesegmentation map. Given the map, separate complexity values may becomputed for individual regions, sets of regions or each of the detectedregions. The resulting multi-valued complexity measure C_(x)(1,2, . . .,M) may then be analyzed to determine the type and amount of enhancementthat will be performed on each region.

When region-specific complexity values are determined, complexity may becomputed differently for separate regions, using different sets offeatures. For example, in a document image, the complexity value forregions of halftone type may be based on the number of text pixels inthe region, while on contone regions it may be determined using anentropy-like feature. It is thus possible to define the most appropriatecomplexity measures for the available region types, and applysegmentation-based enhancements in a more targeted way.

Some region-specific embodiments may be described with reference to FIG.15. In these embodiments, an input image 20 is read and segmentation isperformed thereon 240. This produces a segmentation map 242, which maybe used to measure region-specific complexity values 244. A multi-valuedcomplexity measure 246 results from these measurements. Thismulti-valued complexity measure may then be used to control or adjustregion-specific enhancement 248 of the image.

In some cases, it may be desirable to utilize more than one segmentationmethod when estimating document complexity. One reason for employingmultiple segmentation algorithms is that a single segmentation methodmay not work well for all kinds of input data. For example, asegmentation technique that relies on chromatic information may fail foran achromatic input image. An algorithm that is able to correctlyclassify halftone and text regions on a page but makes errors in contoneareas is not optimal for segmenting scanned photographs. The complexityvalue computed using an inappropriate segmentation method will, in turn,lead to erroneous conclusions about what type of enhancement to apply.

To avoid such problems, an input image may first be segmented using amultiplicity of different segmentation methods. These methods may differin the type of data or features that they utilize, the segmentationapproach they follow, or both. For example, the same segmentationalgorithm may be applied to each of the components of the input imageseparately. In another implementation, the input image may be segmentedinto its constituent regions using a clustering technique, a regiongrowing technique, a fuzzy segmentation method, or other techniques ormethods. After N distinct segmentation maps are generated by thesegmentation module, a complexity value is computed for every map,yielding N complexity estimates for the input image. The bestsegmentation map (and, consequently, the segmentation method) for thegiven input image is then selected based on these complexity values.

Since complexity computations may take into account the errorlikelihoods of the segmentation methods, as well as the document layout,the resulting set of complexity values may provide an indication ofwhich segmentation technique is the most appropriate for the image. Theselection may be done in various ways. For example, the segmentation mapthat yields the minimum complexity estimate can be chosen for furtherenhancement.

Some embodiments of the present invention that comprise multiplesegmentation methods may be explained with reference to FIG. 16. Inthese embodiments, an input image 20 is received and multiplesegmentation methods 252-256 are performed on the image 20. Thesemethods result in the creation of multiple segmentation maps 272-276.Using these maps 272-276, the complexity of image 20 may be determinedin relation to each of the segmentation methods 252-256. Multiplecomplexity values are determined 262-266 corresponding to eachsegmentation method 252-256. These complexity measures are analyzed todetermine which segmentation method 252-256 will perform better for thatparticular image 20. Once a segmentation method is selected, theappropriate segmentation map 258 may be selected and used to perform atuned enhancement 270 of the input image 20.

Alternative embodiments of the multiple segmentation method approach maybe described with reference to FIG. 17. In these embodiments, the inputimage 20 is enhanced using each of the alternative methods and theresulting images are combined to yield a single enhanced image.

Input image 20 is received and processed using multiple segmentationmethods 252-256 thereby creating multiple segmentation maps 272-276,which are then used to measure 262-266 the complexity of the imagerelative to each segmentation method. This results in multiplecomplexity values 292-296 being generated. A tuned enhancement is thenperformed 282-286 on the image 20 according to each of the respectivecomplexity measures 2962-2696. The result of each of the tunedenhancement methods 282-286 is then combined 288 to yield a singleenhanced image.

In these embodiments, all of the maps obtained in the segmentationmodule 272-276 can be used to generate the final enhancement result 288.Tuned enhancement may be performed 282-286 for each of the segmentationmaps 272-276, and the amount or type of enhancement in each case may bedetermined by the value of the corresponding complexity estimate292-296. The enhancement results are then combined 288 to form theoutput image. The final merging step 288 may consider the complexityestimate 292-296 of each segmentation algorithm to determine how eachenhancement result will contribute to the end result. In addition,particular properties of the segmentation techniques can be taken intoaccount during merging, in order to exploit the strengths of eachsegmentation method.

In further embodiments, illustrated in FIG. 18, some of the mapsobtained in the segmentation modules 300-304 can be used to generate afinal segmentation map 330. In these embodiments, the complexityestimates 318-322 for the segmentation maps are combined using afunction F(C_(x)) 324. Among typical choices for the function F(C_(x))are min(.) and mean(.); clearly, other functions may also be used. IfF(C_(x)) exceeds 326 a given threshold σ, global enhancement 328 isapplied to the input image. Otherwise, the segmentation maps arecombined 330 to generate a refined segmentation map 332, andsegmentation-based enhancement 334 is applied to the input image basedon this final map. In some embodiments, only the most reliablesegmentation maps with relatively low complexities may be merged toobtain the final segmentation. An additional step may be added, in someembodiments, where the complexity of the input document is re-estimatedusing the final segmentation, in order to determine whethersegmentation-based enhancement is appropriate.

Many of the exemplary embodiments disclosed above typically relyexplicitly on segmentation maps to estimate the complexity of documentimages. In some embodiments, the segmentation map is not needed tocompute the complexity measure. In these embodiments, other featuresextracted directly from the input image may be utilized. These featuresmay be derived from global image characteristics such as histograms; forexample, smoothness or uniformity of the image histogram can be used asa simple indicator of document complexity. Alternatively, local featurescomputed over pixel neighborhoods can provide the necessary complexityinformation. Such local features may include transform-based attributesdetermined in a block-based manner (e.g., magnitude of high-frequencytransform DCT coefficients), uniformity measures (e.g., local variance),and others. These attributes can then be combined to determine acomplexity estimate for an image. The complexity value may then be usedto determine and adjust the type of enhancement that will be applied tothe document or regions or sections thereof. In some of theseembodiments, the enhancement may not be segmentation-driven; i.e., thesame type enhancement, tuned according to the value of the complexitymeasure, may be applied to the entire document image. Tuning ofenhancement parameters based on C_(x) can be carried out in various waysas explained for other embodiments above.

A variety of document properties and segmentation-related features canbe used to estimate the complexity of a document from its segmentationmap. The particular set of features to be considered may vary accordingto the attributes of the particular segmentation algorithm that is used,as well as the requirements of the application. Certain content typesmay be computationally expensive and difficult to enhance for a givendevice or application; furthermore, enhancement errors committed on someregions may be more noticeable by users and, consequently, significantlymore objectionable.

For example, a device may be able to efficiently process and enhancehalftone areas, but may lack the ability to carry out similarimprovements on continuous-tone regions. For such a device, anappropriate complexity measure may explicitly consider the prevalence ofcontinuous-tone regions in the segmentation map. Additionally, somepixels may be assigned multiple labels during segmentation (e.g., textand halftone for text characters on a halftone backdrop); enhancement ofsuch pixels may be especially hard to handle for enhancement algorithms.Therefore, complexity analysis needs to consider a large number ofcharacteristics of the regions generated by the segmentation method(e.g., shape, area, topological properties, etc.) as possible features.Such features may include, but are not limited to, the number offoreground connected components; the number of non-rectangular connectedcomponents; the ratio of number of halftone pixels to the total numberof pixels; the ratio of the number of halftone pixels in non-rectangularregions to the total number of pixels; the ratio of number of textpixels on halftone to the total number of pixels; luminance and/orchrominance statistics of the background and/or foreground regions, withor without text areas; and so on.

The computed features f_(i) can subsequently be combined in a variety ofways to yield a single complexity measure C_(x) for the entire document.In one embodiment, a weighted sum can be used to generate a scalarC_(x): $C_{x} = {\sum\limits_{i = 1}^{N}{\omega_{i}f_{i}}}$where ω_(i) denote the weights for the features, and can be determinedin a variety of ways (e.g., empirically, through regression analysis,etc.). Other embodiments can use different combination methods to obtainC_(x) from the computed features.

FIG. 19 illustrates the results of an example implementation of documentcomplexity analysis for a sample set. The features used to computedocument complexity in this particular embodiment are the number offoreground connected components; the number of non-rectangular connectedcomponents; the ratio of number of halftone pixels to the total numberof pixels; the ratio of the number of halftone pixels in non-rectangularregions to the total number of pixels; and the ratio of number of textpixels on halftone to the total number of pixels. The feature valueswere then combined using a weighted sum to yield a single scalarcomplexity value C_(x) for the input document. The documents in the testset were divided by human observers into 6 sets of equal complexity, andthe automatically-computed complexity values were compared against thesubjective ordering. As seen in FIG. 19, the computed complexity measureclosely follows the subjective ordering, with discrepancies observedonly at class boundaries.

The terms and expressions which have been employed in the forgoingspecification are used therein as terms of description and not oflimitation, and there is no intention in the use of such terms andexpressions of excluding equivalence of the features shown and describedor portions thereof, it being recognized that the scope of the inventionis defined and limited only by the claims which follow.

1. A method for processing of a digital image, said method comprising:a) performing an initial segmentation process on a non-object-basedimage data set; b) estimating a complexity measure for said image dataset based on the results of said initial segmentation process; and c)performing a refined segmentation process on said image data set whensaid complexity measure meets a criterion.
 2. A method as described inclaim 1 further comprising performing a segmentation-based imageenhancement process using a result of said refined segmentation processwhen said complexity measure meets said criterion.
 3. A method asdescribed in claim 1 wherein global image enhancement is performed whensaid complexity measure does not meet said criterion.
 4. A method asdescribed in claim 3 wherein said criterion represents a condition atwhich segmentation-based enhancement provides no discernible advantageover global enhancement.
 5. A method as described in claim 1 whereinsaid criterion comprises said complexity remaining below a thresholdvalue.
 6. A method for processing of a digital image, said methodcomprising: a) performing an initial segmentation process on an image;b) obtaining process performance data for a refined segmentation processwherein said performance data relates to image complexity; c) estimatinga complexity measure for said image based on the results of said initialsegmentation process and said performance data; and d) performing saidrefined segmentation process when said complexity measure meets acriterion.
 7. A method as described in claim 6 further comprisingperforming segmentation-based image enhancement using a result of saidrefined segmentation process when said complexity measure meets saidcriterion.
 8. A method as described in claim 6 further comprisingperforming global image enhancement when said complexity measure doesnot meet said criterion.
 9. A method as described in claim 6 furthercomprising performing global image enhancement when said complexitymeasure meets an alternative criterion.
 10. A method as described inclaim 6 wherein said criterion represents a condition at whichsegmentation-based enhancement provides no discernible advantage overglobal enhancement.
 11. A method as described in claim 6 wherein saidcriterion comprises said complexity remaining below a threshold value.12. A method for processing of a digital image, said method comprising:a) performing an initial segmentation on a non-object-based image dataset; b) obtaining process performance data for a refined segmentationprocess wherein said performance data relates to image complexity; c)estimating a complexity measure for said image data set based on theresults of said initial segmentation and said performance data; d)performing said refined segmentation process and performingsegmentation-based image enhancement using a result of said refinedsegmentation process when said complexity measure meets a criterion. 13.A method as described in claim 12 wherein global image enhancement isperformed when said complexity measure does not meet said criterion. 14.A method as described in claim 12 wherein said criterion represents acondition at which segmentation-based enhancement provides nodiscernible advantage over global enhancement.
 15. A method as describedin claim 12 wherein said criterion comprises said complexity remainingbelow a threshold value.
 16. An apparatus for processing a digitalimage, said apparatus comprising: a) an initial segmentation processorfor performing an initial segmentation process on non-object-based datarepresenting an image; b) an estimator for estimating a complexitymeasure for said image based on the results of said initialsegmentation; and c) a refined segmentation processor for performing arefined segmentation process when said complexity measure meets acriterion.
 17. A method as described in claim 16 further comprising asegmentation-based enhancement processor for performingsegmentation-based image enhancement based on a result of said refinedsegmentation process when said complexity measure meets said criterion.18. A method as described in claim 16 wherein said criterion comprisessaid complexity measure remaining below a threshold value.
 19. Anapparatus for selectively processing a digital image, said apparatuscomprising: a) an initial segmentation processor for performing aninitial segmentation process on an image; b) process performance datafor a refined segmentation process wherein said performance data relatesto image complexity; c) an estimator for estimating a complexity measurefor said image based on the results of said initial segmentation andsaid performance data; and d) a refined segmentation processor forperforming a refined segmentation process when said complexity measuremeets a criterion; and e) a segmentation-based enhancement processor forperforming segmentation-based image enhancement based on said refinedsegmentation process when said complexity measure meets said criterion.20. A method as described in claim 19 wherein said criterion comprisessaid complexity measure remaining below a threshold value.
 21. A methodfor processing of a digital image, said method comprising: a) performingan initial segmentation process on a non-object-based image data set; b)estimating an initial complexity measure for said image data set basedon the results of said initial segmentation process; c) performing arefined segmentation process when said initial complexity measure meetsan initial measure criterion; d) estimating a refined complexity measurefor said image data set based on the results of said refinedsegmentation process; and e) performing an image enhancement processbased on said refined segmentation process when said refined complexitymeasure meets a refined measure criterion.
 22. A method as described inclaim 21 further comprising performing a global image enhancementprocess when said refined complexity measure does not meet said refinedmeasure criterion.
 23. A method as described in claim 21 furthercomprising performing a global image enhancement process when saidinitial complexity measure does not meet said initial measure criterion.24. A method as described in claim 21 further comprising performing aglobal image enhancement process when said refined complexity measuremeets an alternative refined measure criterion.
 25. A method asdescribed in claim 21 further comprising performing a global imageenhancement process when said initial complexity measure meets analternative initial measure criterion.
 26. A method for processing of adigital image, said method comprising: a) performing an initialsegmentation process on an image; b) obtaining process performance datafor a refined segmentation process wherein said performance data relatesto image complexity; c) estimating an initial complexity measure forsaid image based on the results of said initial segmentation process andsaid performance data; d) performing a refined segmentation process whensaid initial complexity measure meets an initial measure criterion; e)estimating a refined complexity measure for said image based on theresults of said refined segmentation process; and f) performing an imageenhancement process based on said refined segmentation process when saidrefined complexity measure meets a refined measure criterion.
 27. Amethod as described in claim 26 further comprising performing a globalimage enhancement process when said refined complexity measure does notmeet said refined measure criterion.
 28. A method as described in claim26 further comprising performing a global image enhancement process whensaid initial complexity measure does not meet said initial measurecriterion.
 29. A method for processing of a digital image, said methodcomprising: a) performing an initial segmentation process on anon-object-based image data set; b) obtaining process performance datafor a refined segmentation process wherein said performance data relatesto image complexity; c) estimating an initial complexity measure forsaid image data set based on the results of said initial segmentationprocess and said performance data; d) performing a refined segmentationprocess when said initial complexity measure meets an initial measurecriterion; e) estimating a refined complexity measure for said imagedata set based on the results of said refined segmentation process; andf) performing an image enhancement process based on said refinedsegmentation process when said refined complexity measure meets arefined measure criterion.
 30. A method as described in claim 29 furthercomprising performing a global image enhancement process when saidrefined complexity measure does not meet said refined measure criterion.31. A method as described in claim 29 further comprising performing aglobal image enhancement process when said initial complexity measuredoes not meet said initial measure criterion.